DOCUMENT SESUME 



ED 080 543 



TM 003 048 



TITLE 



INSTITUTION 
PUB DATE 
NOTE 

AVAILABLE FROM 



The Assessment of Non-standard programs — the Need 
and the Promise; The Annual Western Regional 
conference on Testing Problems (20th, Oakland, 
California, May 7, 1971).. 

Educational Testing Service, Princeton, N.J. 
7 May 71 
73p. . 

Educational Testing Service, Princeton, New Jersey 
08540 ($2.00) 



EDRS PRICE 
DESCRIPTORS 



MF-$0.65 HC-$3.29 

♦conference Reports; *Educational Accountability; 
Educational Change; *Educational Innovation; 
Evaluation Criteria; Evaluation Methods; 
Instructional Innovation; ♦Program Evaluation; 
♦Testing Problems 



ABSTRACT 

The twentieth Western Regional Conference on Testing 
Problems dealt with the issues of assessment/accountability and 
non- traditional programs. -The following speeches were given: (1) "The 
New Approach of the California State Colleges" by Glenn S. Dumke; (2) 
"Toward Institutional Goal-Consciousness" by Richard E. Peterson; (3) 
"The commission on Non«traditional Study— Who Needs It?" by John A. 
valentine; (4) "The Impact of Mandated Evaluation on Education" by 
Alexander I. Law; and (5) "A Workable Solution to the Demand for 
Accountability: The Georgia Assessment project" by William H. 
Scha backer. A list of conference participants concludes the report. 
(KM) 
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Preface 



The Conference Theme; The Assessment of Non*Standard Programs 
— ^The Need and the Proniise deals essentially with two issues — ^assess* 
ment/accountability and non-traditional programs. The non-traditional 
concept is education's response to demands to do things differently 
and to find new ways of accon:y>iishing what may be old things. The 
papers indicate that the innovative comes more easily than the evalu- 
ative. Suggested departures come from students, legislatures, admin- 
istrators and more grudgingly from the faculty. The character of some 
of these departures seems to be, "anything is possible as long as some- 
one else will do the work.** Glenn Dumke and John Valentine describe 
two major departures from the trad'.donal. 

Richard Peterson describes the complex process of setting goals, 
gaining acceptance of and assessing progress toward them. Alexander 
Law brings us assurance that evaluation maadated by government 
authorities can be treated with sensitivity and humaneness. 

William Schabacker concluded the conference on a hopeful and 
positive note in proposing a workable and working solution to de- 
mands for state-wide assessment. 

Taken as a whole, the papers illustrate the promise which innova- 
tion holds and the complexities of assessing the outcomes of both the 
innovative and the traditional. 

Robert G. Cameron 

CHAIRMAN 
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The Twentieth Annual 

Western Regional Conference on 

Testing Problems 

The twentieth annual meeting of the Western Regional Conference on 
Testing Problems was convened at 9:15 am, Friday, May 7, 1971 in 
the Hilton Inn at the Oakland International Airport. Robert G. 
Cameron, Director of the Western Re^onal Office of the College 
Entrance Examination Board, presided as cbaumao. 



The ''New Approach 
of the California State Colleges'' 

Glenn S. Dumke 

Most of us ii. higher education today, and there are many of us, who 
are considering new approaches to the educational process, are in- 
fluenced, whether we realize it or not, by our own student experience. 
I certaudy am well aware of the fact that my own imdergraduate years 
at Occidental have strongly influenced my concerns about higher edu- 
cation, and my experience also as a young faculty member there left 
me with ideas that I have thought about for many years. The student 
experience produced in me a conviction that comprehensive exami- 
nations for the major were a very good idea. For the first time in my 
life I was faced with an educational program which could not be 
approached in bits and pieces, which bad to be integrated by me into 
a coherent whole, and remembered, so that I could face the evaluatipn 
that loomed ahead. My discipUne, history, was one which encouraged 
this type of intellectual integration, but it became evident to me that 
such stimulation of insights could be applied to almost any field. As a 
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result of having to prepare for a comprehensive examination, the jump 
to graduate work was not a serious problem^ and I feel certain that 
many others have had the same experience. 

Then, when I was a young faculty member at Occidental^ President 
Arthur Coons determined to apply the general education concepts, 
which were then being developed out of Harvard, to our lower division 
curriculum, and in the face of much opposition by a tradition-oriented 
faculty, including myself, he P'lt into effect a "History of Civilization" 
course for five units of credit per semester, lasting two years and in- 
corporating all of the social sciences and humanities plus psychology 
and certain other related subjects. An attempt was made to involve a 
layman's approach to science primarily way of a "history of 
thou^t*' approach, but this was finally deemed unsatisfactory, and a 
concurrent course in science for the non-major was developed. This 
was my first team teaching experience, and, just as when a student, 
I became impressed with the virtues of having one part of a discipline 
rub off against another, I now became equally impressed >^th the 
advantages of having a representative of one discipline bounce his con- 
cepts off others in that and related fields. 

The responses of students to this Occidental experiment were very 
interesting. While they were taking the course they went through the 
usual student process of strenuous objection to the work-load and to 
the number of hours they were forced to spend in class and quiz sec- 
tion, but after they became alumni — and I stayed at the college long 
enough so I was able to get these post-graduation reflections in some 
numbers — ^tiiey were almost uniformly complimentary. Statements 
such as, "This was the finest educational experience I ever had*' and 
**Even though I rebelled against the work-load, I found this was the 
phase of my college years that has stayed with me longest and has 
opened many intellectual doot^" — ^this kind of comment was very 
gratifying and seemed to indicate that the approach was doing its job. 

As the aura surrounding the general education concept died, the 
old faculty pressures began to exert themselves, and gradually more 
and more electives were put into the pfogcsm^ the team-teaching ap- 
proach was diluted, and the argument tiiat it was impossible to decide 
what "every educated person should know*' became dominant. When 
I entered the State Colleges, I found that the general education ap- 
proach had never taken deep root, and although I made some attempts 
to apply h when I was president of San Francisco State, it met with 
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such opposition that I was never able to get very far in this direction. 

The balkanization of the undergraduate curriculum has increased 
since those years, and today there are very few programs which accept 
the principle that a faculty member knows beUer than a student what 
the student should know. I think this is a mistake, and one aspect of 
the proposals I have made to the California State Colleges is designed 
to start a move in the other direction. 

As a result of this background and of many yeai^ of brooding about 
the problem, last January I came before the Educational Policy Com- 
mittee of our Board, wiUi a statement, parts of which I want to quote 
to you now. I started by saying, 

I believe that the time for fundamental changes in the character 
of, and in our approach to, higher education has come. Students 
and faculties and public alike have been questioning the edu- 
cational results of our current systems, and many of us have 
serious doubts about their continuing efficacy. In addition, the 
increasing numbers of persons who want to be educated, and the 
increasing pressures upon the tax dollar, both in California and 
throughout the nation, moke it very clear that under existing 
systems we will either have to limit our service, or thin out our 
operational quality, neither of them very acceptable as alterna- 
tives. The necessary changes will not be brought about by the in- 
flexible, tradition-ridden, Ivy-League-type universities, but rather 
by institutions like ours, young enough to be flexible, historically 
teaching-oriented, and not afraid to try something new. We 
have the opportunity to be the bellwether for the nation in 
changes, certain of which are inevitable. In proposing these 
changes, I am guided by the triple goals of expanding educational 
opportunity for the thousands of students who are knocking at 
our doors; the maintenance and improvement of academic qual- 
ity which we have strug^ed for over the last ten years, and have 
achieved in large measure; and greater value received by both 
students and taxpayers — goals which we clearly cannot hope to 
achieve under our current fiscal constraints and our present rigid 
systems. 



In line with this background statement, I made three basic sugges- 
tions. The first one read as follows: "I propose that we challenge th& 
lockstep, time-serving practice of offering a degree based on the 
accumulation of credits, hours, semesters, and classes attended. I 
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propose that we offer, instead, degrees based on academic achieve- 
ment, carefully measured and evaluated by competent faculties/' I 
stated that I felt the period of time spent in college could be reduced 
by one-half to one full year or more for many, if not for most stu- 
dents, by a deliberately strengthened and advanced placement working 
relationship with the high schools and through comprehensive exami- 
nations given lower-division students. Through such programs credit 
could be given for much of our general education. 

What I was basically proposing was the division of the undergradu- 
ate curriculum into large blocks of work which would then be evalu- 
ated and tested by varying types of examinations worked out by the 
faculties, involving not only existing standardized tests, but also oral 
examinations, creative work projects, etc. 

What needed to be done of course, at the outset, was to determine 
what the undergraduate degree consisted of, and I repeated my own 
concept of a state college degree. Our bachelors degree should repre- 
sent a dual approach, both learning in breadth, which includes the 
development of the perspectives, the problem-solving skills, the com- 
munications competence, and the appreciations of the liberal arts, and 
in depth, which stresses high competence in a specific major area. 
Such a combination of breadth and depth, as fairly well shown by 
existing research, enables a person to live a better life and to adapt 
intelligently to changing conditions, as well as to make a living in 
today's world. This approach stresses human development as well as 
occupational competence. Higher education almost uniformly has 
defined requirements for the degree in quantitative terms, about 124 
to 132 semester units comprising some 2000 fifty-minute periods in a 
variety of courses. The student's knowledge, abilities, appreciation, 
comprehension, and over-all achievements are recognized for degree- 
granting purposes only in bits and pieces. Credits, units, grades, etc., 
signify the completion of the required work of given courses, each of 
which is taken separately, and in many cases dismissed from the mind 
when completed. 

My feeling is that the general education breadth requirements of 
the liberal arts, which comprise a large portion of the lower division 
curriculum, should be redefined in terms of basic bodies of knowledge, 
appreciations, and skills instead of the present definition solely in terms 
of units and courses. The requirements thus redefined might be sub- 
divided into suitable large component parts, each of which would 
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be open to a challenge or comprehensive examination. One possibility 
would be to have general education divided into four areas, social 
science » humanities, science or mathematics, and communications 
skills, each with a suitable challenge examination available. If a stu- 
dent can demonstrate that he can write well, there is no need for him 
to sit through a course of elementary composition. If a student learned 
basic American historj' in high school, there should be no requirement 
which says he must repeat that experience as a freshman. I pointed 
out that ♦her^ currently exist recognized examinations used nationally 
in the areas of general education breadth requirements, and since the 
State Collegci: accept transfer units sight-unseen, we should be willing 
to accept national test scores, particularly if we ^ve the examinations. 

With regard to the upper division, comprehensive examinations 
should also be established in all major fields of knowledge which the 
colleges offer. The objectives should be that a student need not com- 
plete a certain number of units to meet the degree requirements, but 
could instead secure a degree based on a tested achievement level at 
any time he feels competent with the help of faculty advisement to 
subject himself to such an examination. Corollary to this idea was 
that classroom attendance would not be an absolute prerequisite to 
the degree. If the student feels that he can get the information re- 
quired to prepare for his examination by attending class, well and 
good. Classes will be offered. If he is convinced, on the other hand, 
that he can do better by studying in the library or by reading at home 
or by taking an educational TV course or some programmed learning 
device, then these opportunities should be opened to him. It is my 
feeling that we have enough able students so that the pressure on our 
classrooms, which has been intense over the last several years and 
apparently will continue for the next decade, might be reduced some- 
what by this enlargement of varied opportunities for achievement of 
the degree. 

The immediate question arises: does ^lis favor the able student? 
Obviously it gives the able student encouragement to proceed at his 
own pace, which I think is something that is badly needed by our 
cunent educational system. We are losing many students through 
drop-out who are so able that the time-serving approach of the imder- 
graduate cuniculum bores them, and certainly this approach would 
help solve that problem. On the other hand, there are many students 
who need the class work, but hopefully they would be attending classes 
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less well filled, and certainly with some elbow room caused by many 
able students deciding to other methods of acquiring knowledge. 

Obviously these proposals have cost implications which make them 
currently more acceptable than they would have been at other times, 
but the basic reason for the proposal was my own educational philos- 
ophy. The idea is that a cafeteria approach to undergraduate educa- 
tion is educationally and psychologically less satisfactory than an 
approach which forces a student to relate the various parts of his 
educational experience into a coherent whole in his own mind. More- 
over, I strongly feel that if one is convinced that, after a certain date, 
one can forget a subject without penalty, he has less tendency to 
remember it than if he is convinced that he is going to be faced with 
some evaluation nf this knowledge in the future. 

This wa:> the first phase of my proposals. The second phase had to 
do with the concept of the external degree. I pointed out that certain 
aspects of the British "open university" idea might well be applied 
to the California State Colleges, although I stipulated that I did not 
feel that the British approadi or the SUNY approach, both of w4iich 
involve the setting up of a separate institution for the granting of ex- 
ternal degrees, was either suitable or financeable under current condi- 
tions in the California State Colleges. I suggested that we provide 
degree opportunities for substantial numbers of students other than 
through an on-canipus program of students in residence and that we 
start giving degrees through extension. Although the State Colleges 
have a limited extension program, they have never been able to devote 
much attention to the increasingly important field of continuing and 
adult education, because their staff and facilities have been so over- 
taxed with pressures at the undergraduate level. The freeing of the 
undergraduate from required classroom attendance, as I proposed, 
wouid enable some existing facilities to be used for continuing educa- 
tion, and the upgrading of extension classes to equivalency with regular 
academic offerings, together with the ability to mix in a single class 
students on state-support and students on self-support, would also 
open the door to many who could not otherwise be accommodated. 
Moreover, the application of modem technology to higher education, 
such as televised instruction, conespondence courses, self-study com- 
bined with intensive short course on-campus programs, taped lectures 
with study guides, and programmed-leaming, as well as classroom 
instruction on or of! campus could be utilized to extend college oppor- 
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tunities to many more students on a self-support ba^ ise- 
quent reduced demand upon on-campus educatioi ^ cities and 
resources. 

Both of these major proposak involve certain technical and legal 
changes that are basic to their success. One certainly was the necessity 
to get away from present devices for measuring staff and budget and 
to move to a support budget based on a student-faculty ratio with 
different levels of support for lower-division, upper-division, and 
graduate students. In addition, there must be more flexibility in the 
use of faculty time to permit a major shift m the nature of faculty 
responsibility so as to provide g eater attention to advising, counseling, 
and evaluating students. As far as the State Colleges are concerned 
this means a change in the measurement of faculty workload. Currently 
our faculty are measured, and quite rigidly, on their teaching twelve 
hours a week in the classroom. This, together with the required pre- 
paration time, constitutes certainly a full-time load, and in some cases 
more than a full-time load. But if classroom attendance were no longer 
to be required of all students for graduation, then certainly it could 
no longer be required of all faculty for workload measurement. 

In addition, I felt it was necessary to make certain that our exten- 
sion offerings, if they were to be considered interchan^ably with our 
regular programs, and if all classes, both regular and extension, were 
to be attended by a mix of regular and extension students, hi up- 
graded so that they were absolutely consonant in quality with the 
regular program. I am certain that most of our extension program 
fills this requirement now, but certainly parts of it do not. 

In addition, I think it will be necessary to establish ways and means 
of breaking down barriers between and among the colleges if these 
programs are to work. A student should be able to take academic 
work simultaneously at two or more State Collets if they find it 
more convenient or saving in time or more appropriate to their specific 
educational objectives. Arbitrary barriers such as double fees, bureau- 
cratic approval systems, and low registration priorities must be elimi- 
nated. We must become far more flexible in our efforts to fit our 
educational programs to the particular needs of serious students. 

AU of these proposals, of course, are confronted with the hard fact 
that in today's hi^er educational environment the accountability of 
faculty must be rigorously maintained. If classroom teaching is no 
longer to be the basis of workload measurement, then certainly we 
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must develop other auditable means of accounting for faculty pro- 
ductivity. 

I suggested that the way to begin these programs would be through 
pUot approaches, hopefully starting in the fall of 1971. Much progress 
is being made in this direction with the establishment of two task forces 
and a commission on extension education. 

The third major proposal I made was based on an in-depth re- 
evaluation of three areas of our operational practices and programs. 
The first aspect of this had to do with the fact that many of our stu- 
dents take more than the number of units required to graduate. There 
are, of course, a good many reasons for this, many of them quite valid. 
But considering cost factors and other aspects of our current problems, 
I proposed the establishment of an absolute ceiling on the requirements 
for the degree and for majors, and I recommended that we develop a 
fair and equitable system whereby students not pursuing and making 
satisfactory profla'ess toward a degree or credential objective be charged 
the full cost of instruction, and that those who on their own volition 
take work considerably in excess of that required for the degree or 
credential also be charged full costs. 

Secondly, I urged that in spite of the State College record of being 
the most efficient higher educational institution in California in terms 
of facility use, and the fact that California is ahead of the nation in 
this respect, making the State Colleges about as efficient as any insti- 
tution in the nation, we make additional effort to increase the utiliza- 
tion of our existing facilities. This would involve intensified use of late 
afternoon and evening classes and laboratories, and the immediate 
increase of Saturday use. Finally, our data have indicated that certain 
progran:s in our colleges have widely varied costs, and we have not 
been able to figure out why this phenomenon occurs so often. The 
same program in one college will cost far more ihan in another without 
obvious explanation. I proposed a carefu. study o^" the data to deter- 
mine whether costs of some programs could be reduced without loss 
of quality, and whether it was unsound to continue some of them at 
certain colleges. 

These proposed changers certainly cannot be brought about effec- 
tively without the support of the colleges and their faculties, and I am 
very pleased at Uie response I have received from all of our nineteen 
presidents and from many representatives of our faculties and student 
bodies. Many of them cannot be realized without the cooperation of 
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state agencies in modifying the budgeting process in providing greater 
flexibility, and conferences with the Governor and the Director of 
Finance have given me high hopes that we can develop some experi- 
mental flexible approaches to measurement of faculty workload, for 
instance,, so that pilot programs can be started immediately. As a 
faculty member myself, who spent more than a decade in the class- 
room, I look upon these proposals as vastly rewarding to all faculty 
members who are fundamentally concerned with the end product — 
the educated graduate — rather than with the systems, the bureaucracy, 
and the time-worn practices which served well in another day but 
which now are anachronistic* These proposals when realized, hope- 
fully, will enable us to avoid turning away so many students and at 
the same time an emphasis on self-support and extension will enable 
the taxpayer to feel that we do have consideration for him. Finally, 
and very importantly, this will enable the competent and industrious 
student, who is bored and frustrated with our complicated lockstep 
to the point where he often becomes a dropout, to march at his own 
pace, with the challenge of achievement and no wasted time constantly 
before him. 

I concluded my proposals with the following statements: None of 
these proposals, I said, precludes continuing attention to disadvantaged 
or ethnic minorities; in fact, such attention would be facilitated with 
the additional flexibility. In addition, the fact that we are awarding a 
degree based on accumulative and carefully thought-out bodies of 
knowledge and skills which a student must master, not in bits and 
pieces as at present, but as a demanding whole, makes the student a far 
more active participant in the learning process. Instead of sitting in 
large groups to be lectured at with a fixed term of years, much like a 
prison sentence, before him, he will proceed at his own pace, and 
when he has mastered his subjects can be examined on them and be 
evaluated and can then move on. The only limitation will be that he 
will not be allowed to move so slowly that he becomes a burden to 
the state. This to me is genuine education. It puts a premium on indi- 
vidual initiative in the learning process, it frees our crowded class^ 
rooms and laboratories from those who do not need them or who 
because of high ability can move on quickly to other tasks, and it 
opens the door to thousands who want to learn and vAio can pay for 
the privilege but for whom there has been no room in our crowded 
schedules. Such changes, complex and difficult as they are, are neces- 
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sary. Our current systems, effective as they have been in the past, are 
at this moment on the edge of failure. The number of cunent pro- 
posals, which certmnly have influenced my thinkmg m making these, 
are clear evidence of this fact. The impersonality of the large institu- 
tion, the sharp horns of our immediate dilemma between quality and 
numbers, the unuillingness of our clients, the students, to accept much 
longer the rigidities of our present systems, all combmed, force change 
upon us. We should make these changes as they should be made, 
voluntarily and with careful planning, rather than waiting to have 
them forced piecemeal upon us. 

I well recognize that many of these ideas are not ori^nal with me, 
but the package which I have presented to the State Colleges is a series 
of proposals adapted to our particular and specific needs. As I say, 
the response I am receiving is excellent, and we have every hope of 
iiavmg certain pilot programs gping in the fall and the groundwork 
laid for r^id expansion of these new approaches. Once they are tested 
we will decide whether or not to apply them wholesale. I personally 
am very optimistic for their success, and I know that as an educator 
my conscience will be much clearer in terms of our educational end 
product. In any case, what we are domg is giving some of these ideas 
a good try, and what we discover may well be helpful to many of you. 
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I. College Goals In Perspective 

The concept of an "institutional goal'* is just that — a concept, a 
verbal abstraction. But as a conceptual tool, the notion of goals can 
be enormously useful in deliberating, determining, and evaluating 
policy and practice in educational organizations. What should a given 
college or university attempt to accomplish? Educate the able, or 
educate the masses? Teach (tit wisdom of the ages, or prepare youths 
for the job market? Conduct research on any topic for which funds are 
available? Render services to any agency in the corporate or govern- 
ment establishments? Sponsor partisan political action? Sponsor 
ROTC training? These are matters of institutional policy, philosophy 
or ideology. Or, more from the standpoint of contemporary campus 
political realities, whose goals should the institi.:lon embrace — those 
of older, tradition-oriented professors, of research and discipline- 
obsessed faculty, of radical students, of conservative trustees? On 
many canq)uses, these and many more formal and informal interest 
groups hold widely divergent and often conflicting views of the role 
of the institution. What are the implications of such divisions for the 
well-being of the college? Can a modicum of internal consensus about 
institutional mission ever be expected, let us say, at the multiversity? 

Fortunately, all institutions need not respond to the changing times 
in the same way. American higher education is not some kind of mono-* 
lith. Yet diverse colleges — and I suppose I am speaking mainly about 
private colleges — must be able to articulate Uieir unique goals in 
ways that are meaningful to their constituencies, supporters and poten- 
tial supporters, if they are to expect the wherewithal necessary for their 
survival. 

Jacques Barzun has likened the American university to a "iirehouse 
on the comer** that responds to any and all requests for assistance, and 
for many years, with faithful public support, this was a role the uni- 
versity seemed to accept. Institutions simply added new functions to 
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existing ones. This academic bull-market, however, seems to have run 
its course. Financial resources have reached limits of availability. 
Educational costs have risen to new heights, and various external 
a^ncies press the institutions to evaluate their effectiveness and render 
account for expenditure of public and private funds. Yet it seenis that 
demands continue to be made on the institutions to assume new func- 
tions and create new programs. And therein lie the elements of the 
''collision course" in higher education that David Riesman and others 
have warned of — the crunch of new demands against limited resources. 

Let us consider briefly what some of the goals of American colleges 
and universities have been presumed to be, in the past and the present. 

Going way back, the 18th century colleges came into being chiefly to 
educate miniscule elites for positions of leadership in the existing es- 
tablishment. Throughout the 19th century a host of "special interest'* 
colleges were created to serve the interests and values of various 
religious, occupational and social class groups; many of these eventu- 
ally evolved into self-styled "liberal arts" colleges. The great water- 
shed came in 1862 with the Morrill Act; the land grant colleges meant 
publicly supported secular, practical, vocational education for "the 
industrial classes," and they meant public service. Then, toward the 
end of the century, there was the importation of the German idea 
of the university as a center for specialized scientific research and 
scholarship. Perhaps the final major thread is the dramatic rise since 
the end of World War II of the two-year community college, with its 
open doors and community service orientation. 

Thus the conventional wisdom is to ascribe three broad purposes 
to the modem American university.: teaching, research, and public 
seivice. Ph.D. granting universities, however, account for only 300 
or so of the roughly 2600 institutions of higjher education in the 
country. 

Some 350 collets and universities are controlled by the Roman 
Catholic Church. One of the major dilenunas of Catholic higher 
education, according to Andrew Greeley,^ is that many of the Catholic 
colleges are "seeking the same objectives as the rest of American 
higher education seeks, (while) also pursuing objectives which are 
uniquely their own/' Thus, an excerpt from one coUe^ catalogue: 

It is the aim and purpose of college to assist students in 

the attainment of the highest perfection of intellect and will of 
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which they are capable, in order that their earthly life may be 
spent in the service of God and man, and their eternal life in the 
blessed and complete happiness of union with God in heaven. 

Some 450 colleges are affiliated with one or another of the Prot- 
estant denominations. Strength of the ties varies greatly from college 
to college, ranging in reli^ous stance from tightly fundamental to 
highly liberal. While the clear trend over the years has been toward 
a weakening of denominational ties, many continue to "keep the 
faith." A catalogue excerpt: 

The founding ideal of is to provide young men and 

women of the twentieth century the opportunity to hivcstigate 
truth from the position that all areas of true knowledge and 
divine revelation are compatible. 

It is not easy to do justice to the rhetoric of "liberal arts education.** 
Much of the more recent outpouring may be a natural response to 
the somewhat embattled condition of the liberal arts — ^under attack as 
it is by populist and vocational forces, advanced programs in the high 
schools, and pressures for graduate preparation and academic pro- 
fessionalism. Indeed, Jenks and Riesman spenk of the "univereity 
college,** as they call it, the college that prepares p^ple for graduate 
school, as the key consequence of what they rjill the "academic revo- 
lution.*'2 All this said, the goals of liberal mis colleges are commonly 
couched in terms of mastery of a basic intellectual heritage together 
mih development of intellectual values and styles, aesthetic sensitivity, 
and attitudes of social and moral responsibility. 

The scores of public four-year colleges across the country, while 
giving lip service to liberal arts purposes, are primarily in tne business 
of vocational and pre«professional training, especially of teachers. The 
purposes of the public junior colleges, of which there currently are 
some 800, enrolling a third to a half of all the freshmen and sopho- 
mores in the country, generally involve providing terminal technical 
and vocational training, the firsi two years of college for students 
transferring to four-year institutions, and a range of public services 
for individuals and agencies in the local community. 

Finally there is a variety of specialized institutions, such as technical 
institutes, theolo^cal schools, and art colleges, whose purposes are 
more narrowly drawn: 
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The primary purpose of the undergraduate school of , 

as stated by the Trustees, is **to provide a collegiate education 
which will best train the creative type of scientist or engineer so 
urgently needed in our educational, governmental, and industrial 
development." 



il. Some Institutional Uses Of Institutional Goals 

Let me move on now to outline several ways that clear conceptions 
of institutional goals may be put to use on the campus. Some of the 
uses, such as the first two, are fairly general; the others are more 
specific. This listing is certainly not exhaustive, and the various entries 
are not independent either in the abstract or in practice. 

(1) As fundamentals of policy. A conception of institutional goals 
may serve as the basic element in a formulation of the institution's 
policy, philosophy, or ideology. Stated goals help to tie together as- 
sumptions, values, and hopes for the institution into a coherent policy 
that then provides standards for present and future college operations. 

A policy formulation containing clearly enunciated goals also 
enables individuals and agencies external to the campus — prospective 
students and staff, ^vemmental units, funding agencies, for example 
— ^to be clear about the college's reasons d'etre and what can be 
expected of it. 

(2) As general decision guides. A policy-as-goals statement, es- 
pecially if democratically conceived and widely understood in the 
college community, should serve the entire community as a frame** 
work for reaching decisions, solving problems^ allocating resources, 
and accordingly ordering actions in certain directions and not in 
others. The goals can be used as decision standards by all campus 
groups: by the trustees, for example, in approving architect's plans 
for die new student union, by department chairmen in recruiting 
faculty^ by students in considering revisions to the judiciary code, 
and so forth. 

(3) In planning. As institutions and systems have had to cope with 
expanding enrollments, and, now, wiih limited finances, they have 
been forced to engage in some sort of planning, be it crude or fairly 
systematic, short or reasonably long-*term. The importance of goal** 
setting at the outset of the planning process is universally emphasized 
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by professional planners in both educational and noneducational set- 
tings. 

Planning in higher education, of course, goes on at many levels and 
goal consciousness, it may be argued, is critical at all of them: m the 
most futuristic thinking about national and international systems, in 
planning Siwash's next five years, in year-to-year budgeting in sb-^e 
institutions and their component units. 

lliis last is particularly important for purposes of this paper. In the 
past few years there has been a notable infusion into higher education 
of various public finance analysis and management methods, of which 
perhaps the best known goes by the letters "PPBS." An important 
element in almost all PPBS and related models is identification of 
goals or "outputs" (the economists' preferred term). Vario'^s planners 
on college campuses who have written about the mattei, however, 
pointed to the very great difficulty, in practice, of developing usable 
conceptions of college goals. 

(4) In management information systems. A response chiefly to 
increasing university size and complexity, the management information 
system (MIS) is another new administrative tool cunently enjoying 
a considerable vogue. MIS's have been developed to provide decision 
makers with relevant and timely ("computerized") data, use of which 
presumably leads to better decisions. As with the more general plan- 
ning process, "a management information system calls for the clear 
explication of objectives and expose of the processes by which the 
objectives are reached."^ 

(5) In institutional evaluation. The field of educational evaluation, 
as this audience hardly needs to be told, has grown into a new pro- 
fessional specialty with an evolving set of principles and techniques 
all its own. Evaluation has come to be understood as a process of 
information gathering focused on the extent to which an educational 
program is achie^dng predetermined objectives. Evaluation informa- 
tion is passed along to educational managers either (or both) during 
the course of the program or at its termination; in either event, the 
purpose is program improvement, meaning maximization of program 
objectives. 

For the most part, educational evaluation has occurred in elemen- 
tary and secondary schools and has been focused on specific courses 
or programs. Evaluation, however, can conceivably be extended to 
cover an institution's total educational program, and it is beginning 
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to take hold in higher education. Many iiniversities have institutional 
research offices; there is a nationally organized Association for Insti- 
tutional Research (AIR); a number of consortia of colleges have 
been formed to promote cooperative institutional r^arch; and a 
range of assessment instruments have become available. 

(6) In implementing accountability. "Accountability" is another 
concept sweeping across the educational landscape, especially, so far, 
in lower rather than higher education. Leon Lessinger, late of the 
U. S. Office of Education, and perhaps the father of the concept, has 
said:^ 

In its most basic aspect, the concept of educational account- 
ability is a process designed to insure that any individual can 
determine for himself if the schools are producing the results 
promised. 



He goes on to say: 



Like most processes that involve a balancing of inputs and out- 
puts, educational accountability can be implemented successfully 
only if educational objectives are clearly stated before mstnic- 
tion starts. 



The distinction between evaluation and accountability implemen- 
tation is not entkely clear, at least to me. Accountability seems to be 
concerned more with end results and less with process or means, has 
more a financial and efficiency focus, is more of a public operation 
(like an audit by an external agency), and carries a greater implication 
of finality — of hard judgments about total programs (rather than of 
trying to unprove on existing on^). While prospects for this sort of 
accountability may seem distant for most colleges and universities, it 
seems to me the writmg is on the wall. 



ill. ET5 Research on Institutional Goals 

What I am going to do now is lay out for you an R&D saga in three 
chapters. The first is of the paft, completed; the second is underway 
right now; and the third is on the drawing boards. 

Actually there is also a kind of introduction to the epic. Three to 

16 



Richard E. Peterson 



four years ago several of us in Princeton — ^Pat Cross, notably — began 
thinking about the need for institutional goal definition, mainly m the 
context of evaluation. Any viable model for evaluating the effective- 
ness of a college, we thou^t, had to start with the institution identify- 
ing its goals. At any rate, in late 1969 — and now we are into Chapter 
I of the epic — an opportunity presented itself in the form of a grant 
from ihe Regional Educ&acn LalM)ratory for the Carolinas and Vir- 
^nia (RHLCV) for a study aimed at defining the goal structures of 
five colleges that were working with the Lab m developing its Admm- 
tstrative-Organization System (AOS) model. Norman Uhl, then of 
L'TS's Southeastern Office, was the project director— and Uhl set 
himself the task of testing out what is known as the Delphi technique 
in achieving consensus among diverse campus constituent groups re- 
garding the goals of each respective institution. Thus, the objectives 
of the project were, first, to test the usefulness of the Delphi technique 
as a way of obtaining consensus about goals, and, second, to learn, for 
purposes of institutional self-study, how diverse constituent groups, 
on and off campus, perceive the goals of the respective colleges. 

So, what is the Delphi technique? Briefly, it involves the following 
four steps: 

(1 ) participants are asked to list their opinions on a specific topic, 
such as recommended activities or predictions for the future; 

(2) participants are then asked to evaluate or rate the total list 
against some criterion, such as importance, chance of success, 
etc.; 

(3) each participant receives the list and a summary of responses 
to the items and, if in the minority, is asked to revise his 
opinion or indicate his reason for remaining in the minority; 

(4) each participant again receives the list, an updated summary 
of responses, a summary of mmority opinions, and a final 
chance to revise his opinions. 

Thus, applied to the matter of college goals, the Delphi method 
has the potential for providing an institution with (1) a range of 
ideas about goal;, (2) a priority ranking of the goals, and (3) a 
degree of consensus about goals. 

In Uhrs study, the major departure from the standard Delphi 
procedure was to omit the usual first step of asking respondents, in open- 
ended fashion, to list ideas. Instead, step one consisted of adminis- 
tering a previously prepared experimental Institutional Goals Inven- 
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tory (IGI). Figure 1 is an excerpt from that first inventory. The 
items were written by a group of ETS psychologists and sociolo^sts, 
under the general direction of Uhl, in January 1970. This preliminary 
IGI was distributed to some one thousand individuals spread across 
samples of undergraduates, graduates (where applicable), faculty, 
administrators, trustees, and alumni from the five institutions, plus a 
small cross-section of people in the local community. The instrument 
consisted of 105 statements covering the 18 kinds of goals listed across 
the top of Figures 2 and 3, 

Respondents rated each item on a five-point "importance" scale, 
and each item was rated in terms of both ( 1 } perceptions of the exist- 
ing goal structure, and (2) what the institution's goals ought to be 
(i.e., they gave "is" and "should be" responses). Eighty-five percent 
of the questionnaires were returned. 

The second step was to distribute the same form to the same one 
thousand people, with two differences: the first was that the modal 
(most frequent) "is" and "should be" responses for each item were 
indicated on the form; and, second, individuals who this second time 
assigned a rating different from the step one modal rating were asked 
to write out briefly the reasons for their rating. Return rate for the 
second questionnaire was 80 per cent. 

The third step was a repeat of the second, with the exception that 
this time separate sheets containing a summary of the minority 
opinions for each goal statement for the institution in question accom- 
panied the inventory. Thus, in step three, participants responded to 
the IGI, knowing, for each item, both the modal response on the 
previous administration and the kinds of reasons people had for not 
giving the modal response. Return rate: 75 percent. 

A small sample of the results is presented in Figures 2 through 5. 
Institution A is a church-related university located in South Carolina. 
What is noteworthy about the Figure 2 profiles is their similarity — 
how close together they are. It is interesting to speculate about what 
this means. Does it mean satisfaction? Does it mean complacency? 
Does it mean the end of aspiration? 

Figure 3 depicts a predominately black university in North Carolina. 
Of the five institutions in the study, this was the one with the largest 
discrepancy between the **is" and "should be" profiles. The differences 
must mean that people are dissatisfied. Yet I think they also mean that 
there is a large measure of aspiration, that people want to move in a 
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great many directions (ejxept towards a religious orientation). 

Figure 4 illustrates an instance of Delphi-"encouraged" goal con- 
vergence — specifically, regarding "National and lu'emational Service" 
(as an institutional goal) at the aforementioned South Carolina uni- 

Figure 4 

Institution A: Plots of Constituent Group Means for Three Questionnaire 
Administrations "Is'* Ratings for National d International 
Service Goals (from Uhl, 1971) 
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versify. The letters represent constituent groups; "P" stands for Par- 
ents, "G" for Alunrni, and so forth. The three clusters correspond to 
the three successive questionnaire administrators (Ql, Q2, Q3, across 
the bottom). On the first administration, the eight groups were quite 



Figure 5 

Institution C: Plots of Constirurnt Group Means for Three Questionnaire 
Administrations "Should Be** Rating for Freedom Goals (from UM, 2971) 
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far apart. On the second, the plots were closer, and on the third they 
were practically together (only two groups slightly deviant). 

Figure 5 shows an instance where very little convergence took place. 
Institution C is a liberal arts college in Virginia. Freedom, as a goal 
area, involved academic freedom, personal freedom, allo>^ing people 
to live their own lives, and so forth. Students and faculty did not shift 
at all from the first to the third questionnaire. The trustees moved 
somewhat. With low scores indicating high importance, the students 
(naturally) attached the greatest importance to freedom on the cam- 
pus, and the trustees, the least. The Freedom and Religious Orienta- 
tion goal areas consistently showed the least convergence at all five 
colleges; certain fundamental moral convictions seem to be relatively 
immune to Delphi influence. 

In looking over all the Delphi plots (18 goal areas, **is" and "should 
be" ratings, five colleges), Uhl and I were impressed by the large 
number that showed definite convergence. Figure 4 is a selected case, to 
be sure; yet some three-quarters of all the plots depicted clear and sub- 
stantial convergence. (To what extent Delphi-induced changes signify 
permanent attitude or behavior changes is probably an open question.) 
In gener::!, off-campus constituent groups shifted more (toward the 
on-campus groups) than did the campus groups, especially on the 
**is'' ratings, reflecting, I suspect, a fairly rational deference to the 
greater knowledgeability of the on-campus groups (faculty, adminis- 
trators, etc.). A final clear finding from Uhl's study^ was that con- 
siderable convergence took place within constituent groups, as well 
as between them. Stmidard deviations were almost invariably lower 
on the third than on the first questionnaire administration for a given 
constituent group on a given goal area. 

The second chapter in this saga began just last February (1971), 
when some decisions were made to the effect that ETS would move 
fairly quickly to develop a goals inventory to be made available to 
colleges and universities in the fall of 1971, this coming fall. Norman 
Uhl had left ETS to return to university teaching and research, and 
the job of working with the IGI fell to me. 

First of all, it was pretty clear that we couldn't market an instrument 
that had only been tried out at five institutions in the Carolinas and 
Virginia. In March, then, we began organizing a small pretest of a 
revised IGI, to take place in the West in May. Not much lead time, 
and May is probably the worst possible month to try to arrange for 
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students and faculty to fill out questionnaires. We invited a dozen col- 
leges to participate, hoping to get four, we ended with ten. 

Also in March, working^ with Uhl as a consultant, a number of 
additional analyses of the RELCV data were carried out. We at- 
tempted to determine whether items clustered together so that we 
could make decisions about which items from the preliminary IGI 
should be relevant for the new inventory. Included were four factor 
analyses, item intercorrelations for "is" and "should be" ratings— 
separately for students and faculty, as well as item means and standard 
deviations for these four groups. 

We, Barry Morstain* and myself, began working with these data 
on April 5. We began cUnunating items from the oripnal instrument: 
items that were highly correlated, since we wanted every item to yield 
essentially unique information; items that were highly skewed or for 
which there was little response variation; items that showed little 
difference between the mean "is" response and the mean "should be" 
response. 

At the same time we were working toward a slightly different con- 
ceptualization than the one embraced by the preliminary form. An 
Altruism/Humanism category was added — in part as a supplement to 
the Traditional Religiousness category, and an Accountability/Effici- 
ency cluster of items seemed appropriate to the times. In addition to 
providing a focus for item writing, a conceptualization such as this 
one (Figure 6) serves at least two purposes: first, it provides a theo- 
retical description of the domain — in this instance the domain of 
college and university goals — that the instrument is intended to meas- 
ure; second, it has the more practical purpose of suggesting ways of 
scoring groups of items togeflier, as scales or indices, which in turn 
make for convenience in summarizing and interpreting the results of 
the inventory on the canq)us. By this last, I mean it is often advan- 
tageous to be able to report research results in terms of 20 or 22 scale 
scores rather than in terms of the frequency distributions on 100 or 1 10 
individual items. 

Scale scores would also be more reliable than the responses to 
individual items. People at ETS tend to put great store on test relia- 
bility; they build long tests and obtain reliabilities in the high .90's. 



* Acting Director, Academic Planning and Evaluation, University of Delaware, 
Newark, Delaware. 
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Thus the College and University Environment Scales (CUES), au- 
thored by Robert Pace at UCLA, has a small number of fairly long 
scales mih high reliaWIities. A more recent mstrument from ETS, the 
Institutional Functioning Inventory, contains eleven 12-item scales 

Figure 6 

A Tentative Conceptualization for the Revised 
Institutional Goals Inventory (IGI) 

Output Goals 

1. Academic Development (acquisition of knowledge, academic mastery, 
etc.) 

2. Intellectual Orientation (as an attitude, style, commitment to learning, 
etc.) 

3. Individual Personal Development (of one's unique human potential, 
etc.) 

4. Humanism/Altruism (idealism, social concern, etc.) 

5. Cultural/ Esthetic Awareness (appreciation, sensitivity to the arts, etc.) 

6. Traditional Religiousness 

7. Vocational Preparation 

8. Advanced Training (graduate, professional) 

9. Research 

10. Meeting Local Needs (community public service, etc.) 

11. Public Service (to regional, state, national, international agencies) 

12. Social Egalitarianism (meeting educ. needs of people throughout the 
social system) 

13. Social Criticism/ Activism (toward change in American life) 

Support Goals (internal college goals intended to help realize the "output" 
goals) 

14. Freedom (academic, personal) 

15. Democratic Governance (emphasizing structural factors) 

16. Community (emphasizing attitudinal factors — ^morale, spirit, ethos) 

17. Intellectual/Esthetic Environment (intellectual stimulation, excite- 
ment, etc.) 

18. Collegiate Environment (extracurricular activities, social life, athletics, 
etc.) 

19. Innovation 

20. Evaluation and Planning 

21. Accountability/Efficiency 

22. External Relations (toward understanding and mutually beneficial re- 
lations between campus and external constituencies) 
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with reliabilities averaging about .90. In building the IFI, we followed 
the customary item-analyses procedures designed to maximize internal 
consistency reliability. We got good reliability, from faurly short scales, 
but at some cost in making every item work — ^yield unique information 
for the user college. Items in a given IFI scale tend to intercorrelate in 
the .70's. With the IGI we will be covermg a broader conceptual 
domain [22 goal areas], with shorter and less reliable scales, but every 
item will be doing work — providing unique information to the college. 
I expect that the five items in a given IGI scale will intercorrelate 
about .40 on the average. IGI reliabilities will not be ridiculously low; 
Uhl obtained coefficient alphas in the .70's with four, five and six item 
measures. 

Once we were satisfied with the modified conceptual framework, 
and having decided that each scale would consist of five items, we 
began formulating new goal statements — entirely new sets of items for 
the new constructs such as Accountability/Efficiency and Altruism/ 
Humanism, and additional items to round out the existing categories 
where not enough of the old items survived the various statistical 
criteria. By mid-April we had a long draft inventory for sharing with 
colleagues in Berkeley and Princeton. The following week was spent 
haggling about phraseology and generally worrying the items into 
forms we were satisfied with. (Parenthetically, many people look 
down on item writing as a menial task, something you turn over to 
research assistants. I personally think that writmg questionnaire items 
that really do good work for you is a fairly challenging intellectual 
task.) Also during that week we decided that "Joe College" was still 
alive on many campuses, and that we indeed needed items about 
bigtime athletics, fraternities and Uie like — i.e., a Collegiate Environ- 
ment scale. Then, at the last minute, on the advice of friends in Prmce- 
ton, we went from what was a sin^e "intellectual development'* cate- 
gory to the twin scales of Academic Development and Intellectual 
Orientation. The eventual revised IGI, then, consists of 110 goals 
statements— five for each of 22 goal categories (see Figure 6). 

%)uring May, right now, the colleges are distributing the form to 
samples of 100 or 150 students and faculty. One college is also in- 
cluding their trustees and another its administrators. Still another is 
planning a fairiy large administration to alumni 2nd parents. We will 
do the scoring in Berkeley in June and send back the colleges item 
tabulations and mean scores for the 22 scales, fc. \ h the "is" and 
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"should be" ratings, separately for students, faculty, and trustees. 

Then in July we will organize a comprehensive review of everything 
done thus far. We m\\ consider the conceptualizations, items, and hard 
data from both the RELCV and west coast studies. We will look at 
the "soft data" — the critical comments from respondents and the new 
ideas about goals offered on the last page of the inventory. More 
important, we will want to make use of one or more panels of in- 
formed and insightful people from the campuses, who can hel ) us 
insure that the instrument, insofar as possible, covers the domain of 
institutional goals for the broad spectrum of American higher edu- 
cation. More important than any of the statistical criteria, it seems to 
me, the instrument must deal with issues that colleges are struggling 
with as they formulate and modify institutional policy and practice. 
Such issues and goal conceptions, of course, are constantiy changing 
and evolving. I would hope, myself, that any operational IGI would 
assume revision, perhaps on a yearly basis. 

At any rate, from out of this comprehensive review will come a final, 
operational IGI, printed m machine-scorable format, and distributed 
and scored through ETS's Institutional Research Program for Higher 
Education (iRPHE), the program that distributes the College Student 
Questionnaires (CSQ), CUES, the IFI, and the other instruments and 
services for institutional self-study. 

Let me quickly outiine tentative plans for the next one to two years 
— the third chapter in this stirring story. 

When the IGI becomes available in the late fall, there will be no 
norms for the instrument, no comparison data against which a given 
college can interpret its own IGI data. The plan is to carry out a 
national norming study during the 1971-1972 academic year, with the 
cooperation of a sample of perhaps 100 colleges and universities. It 
will be a stratified rather than a random sample. We will want to 
have, say, 10 each of public universities, private universities, Prot- 
estant colleges, public junior colleges, and so forth, so that separate 
norms can be assembled for some ten different types of institutions. 
At each institution the form will be administered to samples of up to 
150 faculty, perhaps 200 to 300 students, and to all the membei^ of 
the governing boards. Each set of norms (e.g., for public universities), 
then, will consist of a rather substantial amount of data — item and 
scale norms, for both the "is" and "should be" responses, for faculty, 
students and trustees. 
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During the following year a manual for the use of the inventory will 
be prepared, which, in addition to most of the usual kinds of informa- 
tion found in test manuals, will also contain an extensive discussion of 
how a college might make use of the IGI data in setting goals and in 
otherwise reaching decisions and drawing plans in various areas of 
institutional policy and practice. 

The next two years' work, however, will represent more than just 
a norming study. If all goes as anticipated, the project should also be a 
major substantive study of purposes in American higher education — 
in particular, of how people at different types of colleges across the 
country understand the goals of their institutions — both as they per- 
ceive them now, and as they think they ideally should be. Furthermore, 
I am hoi»ng that we can study what various groups of people off the 
campus believe about tiie goals in higher education in America. It 
should be possible, perhaps with the cooperation of graduate students 
at the state university in, say, six state capitals, to administer the IGI 
to people such as state legislators, high school teachers, business 
leaders, construction workers, policemen — ^with the local public uni- 
versity as the institutional referent. 



IV. Conclusions 

(1) Perhaps it is gratuitous to say that the college intending ser- 
iously to redefine its goals must first consider whether or not it indeed 
has the power to define its own directions and then to act in pursuance 
of such new understandings. I say this mindful of the clear trend of 
more and more colleges to find themselves deferring to higher authori- 
ties. This question of autonomy is particularly unavoidable in the pub- 
lic sector, with the rise of statewide systems, coordinating bodies, and 
master plans, together with seemin^y hardening orthodoxies about 
v/hat certain kinds of colleges are supposed to do — about the role of 
the public junior college, for example. 

Should all the campuses in a system be similar or "comparable," or 
should each strive for distinctiveness? There has to be coordination 
in a multi-campus system, no doubt about it. Yet, there also has to 
be, I am equally certain, opportunities for meaningful participation 
by the people involved in the educational work of the campus, in 
determining the content and process of that work. Reaching accommo- 



29 



institutional GoahConsciousness 

dations on these and related issues will require administrative states- 
manship of the highest order. 

(2) Institutional goal determmation, it seems to me, has two end 
products: (a) identification (statement) of goals, and (b) establishment 
of priorities among the goals. An institution's "goal structure* —its 
rank-ordering of goals — can be said to be "determined*' when some 
level of consensus has been reached through a process that is demo- 
cratic and participatory. The goal determination process must univer- 
sally (on the campus) be regarded as fair if the resulting goal structure 
is to have legitimacy, if it is to be accepted as morally proper in the 
college community. These are heavy problems, elaboration of which is 
well beyond the scope of this talk. 

(3) Whatever the specific goal determination mechanisms adopted 
may be, the responsibility for setting the process in motion, for laying 
out the charge, and for dealing with the autonomy question, lies with 
the chief campus administrator. This is the conclusion of a number of 
people who have studied the situation rather more ctosely than I have. 
Determination of college mission, in short, is a critical leadership 
function of the college president. 

(4) Institutional goals would profitably be conceived in two cate- 
gories: 

[a] Outcome goals. These are the desired states the college seeks 
to realize — characteristics of graduating seniors, kinds of research 
and development to engage in, kinds of public services to perform, and 
so forth. These goals, I should think, would be stated at about the 
level of specificity of the goal statements used in the two studies I 
mentioned. 

[b] Support goals. These are the goals, attainment of which facili- 
tates reaching the outcome goals. They have to do with instructional 
measures, educational environment, and the like. In a sense, they are 
planning goals (e.g., of a five-year plan) : to double the library hold- 
ings, or the number of fine arts faculty; to establish a center for eco- 
logical studies or a remedial skills center, for example. Support goals, 
in short, are intended to optimize the previously identified outcome 
goals. 

(5) The mechanics of institutional goal determination might well 
involve both a committee*like task group and some form of opinion 
or values survey, such as the IGL The ta&k group should include repre- 
sentatives of the various campus constituent groups, including trustees 
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(who presumably have encouraged the goal analysis effort from the 
start). Task group members could be elected by their respective con- 
stituencies, or they might be volunteers. Institutions having an All- 
College Assembly or some other unicameral governing body could 
form a goals determination subgroup from the membership of the 
unicameral body. 

Numbering about twelve members and chaired by the college presi- 
dent, an important job of the Task Group on Goals is to organize, 
help plan and implement, and generally oversee a goals survey. Once 
the survey is completed, the Task Group would conduct open hearings 
on the survey results, and eventually prepare a report setting forth a 
goals structure for the college. 

(6) Finally, it seems essential in these times that colleges articu- 
late their goals — to give direction to present and future work, to 
provide an ideology that can nourish internal cooperation, communi- 
cation and trust, to enable assessment of the institution as a means- 
ends system, to afford a basis for public understanding and support. 
Indeed, the college without the inclination or will to define itself, to 
chart a course for itself, can look forward either to no future, to a 
kind of half-life of constantly responding to shifting pressures, or to a 
future laid down by some external authority. Neither prospect pleases. 
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The Commission on Non-traditional study is in the early springtime 
of its life. It has roots. Its roots are in an apparently hospitable soil 
and the weather is favorable. Above ground it has a stem and rudi- 
mentary branches. Its fruits, by which it will be known, are in the 
future, however. At this time one can only speculate, anticipate, and 
hope. \ 

About its roots. Five fibers are prominent in the root structure. 
Their names all happen to start with C. The first is James B. Conant, 
fomier president of Harvard University and ambassador to West 
Germany, and author of books on American secondary school educa- 
tion. He invented Educational Testing Service, or by his own more 
modest account (in his autobiography, sub-titled Memoirs of a Social 
Inventor) was at least instrumental in promoting the idea and plan 
for the merger of testing programs on which ETS was based. 

The second is Henry Chauncey. During his long and productive 
tenure as president of lETS he was an early, vigorous, and persistent 
advocate of credit by examination at the college level. 

The third is the Carnegie Corporation of New York. It has long 
supported mih vision and funds the idea of alternative avenues for 
college study, credit and degrees. It has given generous support to the 
College Level Examination Program. Alan Pifer, president of Carne- 
gie, has taken the lead in calling attention to the possibilities of external 
degree arrangements in this country. Came^e recently joined with the 
Ford Foundation in making grants of one million dollars to the State 
University of New York, for the development of off-campus programs 
of study leading to degrees at colleges within the SUNY system; and 
eight hundred thousand dollars to the New York State Education 
Department for the development of New York Regents degrees, based 
on examinations. The Commission is supported by a grant from 
Carnegie of $140,000. 

Fourth is the College Entrance Examination Board. The College 
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Board has sponsored the Advanced Placement Program, which has 
helped many college freshmen start out with credit for college-level 
high school study. It has more recently developed the College-Level 
Examination Program. It has also sponsored a series of Commissions 
— ^the Commission on Mathematics, the Commission on English, and 
most recently the Commission on Tests — ^which provide traditions and 
experience for this new commission. 

Finally, I feel compelled to mention candidates — candidates by the 
millions who have taken the CEEB Scholastic Aptitude Test and 
Achievement Tests, and by paying for the privilege, pleasure or pain 
have provided much of the cash for the fadlities, staff and good works 
of the College Board and ETS. 

It was just about fifteen months ago, at a joint meeting of the Col- 
lege Board and ETS executive committees, that it was decided the 
two organizations should explore how they might advance and support 
mor£ widespread opportunities for college-level study and its recog- 
nition. Jack Arbolino, on the College Board staff, and John Valley, on 
the staff of ETS, were commissioned to study the possibilities. They 
produced background papers, and were led by their own findngs and 
deliberations to propose the formation of a national university, which 
they felt would benefit colleges as well as a wide variety of incfividuals. 
With the Arbolino-Valley papers as stimulus, the Board and ETS 
officers and trustees eventually settled on a joint, two-part approach. 

One part involved the establishment of an Office of External Degree 
Plans, to serve as a link between both the Board and ETS, on one 
hand, and on the other hand, institutions and agencies involved in 
developing external degree programs and interested in the instruments 
and services the Board and ETS are able to provide or to develop. 
The Office of External Degree Plans is now in being. John Summer- 
skill will serve as Director of this office, starting next fall, and in the 
meantime George Hanford, Executive Vice President of CEEB, is 
serving as Acting Director. 

The other part involved the creation of a Commission on Non- 
traditional Study, also to be sponsored jointly by the College Board 
and ETS, but to serve independentiy, in the public interest, as a body 
designed to consider from a national perspective the full range of 
issues raised by the expanded use of non-traditional forms of study 
and its recognition, including external degrees. 

A proposal for such a Commission was submitted to Carnegie in 



33 



Commission on Non-traditional Study 

December; the proposal was accepted and funds granted in January; 
the Commission was appointed in February; and it met for the first 
time, in Sarasota, Florida on March 8-9. At this meeting, chaked by 
Commissioner Samuel B. Gould, the Commission made substantial 
progress in clarifying its charge and the scope of its concern; also in 
organizing itself for the work ahead. 

The soil for the Commssion. There is little doubt that the soi^ 
must be fertile these days for the growing of a lush variety of options 
for getting college credits anH degrees, for learning acquired inside and 
outside of classrooms — ^Wit^Aim the walls and outside the walls of 
colleges — in this country and also abroad — ^books, TV, computers 
and tapes — at work and out in the community — from service in the 
Armed Forces, on land, sea and in the air. One has to only look at 
what is going on here in California, in New York, and in many other 
states — at dozens of colleges, including those participating in Uni- 
versity Without Walls — in other countries, notably England, with its 
Open University — and in other likely and unlikely places. 

There are mrny answers to the question why all these developments 
are occurring. In general, there is a large and growing gap between the 
goal of educational opportunities after high school for all who need 
and want them, and the facilities available to achieve this goal. There 
are millions of Americans, such as housewives, workers, older people, 
and the very poor, for whom no real opportimities are available. There 
are many others, including some of college age, for whom opportuni- 
ties are available but fall short of satisfying the real educational needs. 
There are many colleges, univenities and systems confronted with 
the challenge of educating more students more successfully, with re- 
duced budgets. 

There is spreading within the - ublic a sense that the old, familiar 
academic ways obstruct the development of student minds, talents and 
personalities. 

It is one tiling to analyze the soil for the nourishment it provides to 
programs of non-traditional study or credit. It is another thing to 
analyze its power to nurture and sustain a Conmiission set up to 
influence such programs. This power is not so certain. To be sure 
Commissions seem to grow like weeds in this country — ^like weeds many 
chok, ) however, live and die having served no apparent purpose. From 
the response to the Commission so far it is clear that many look hope- 
fully to it for wise guidance and productive stimulation. This is en- 
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cc/raging, and also very challenging. 

A -word about the Commission's Morphology* Above the surface 
the basic form of tf a Commission may be observed, with some features 
quite distinct, and others rudimentary. 

First, there is the Commission itself.* The 27 members include the 
heads of private colleges and of public institutions and systems. Four- 
year and two-year institutions are represented. There are members 
from the worlds of continuing education, ac .Station, educational 
technology, libraries, labor, government, and educational associations 
and institutes. They come from all major quadrants if not all corners 
of the United States, with Washington, D. C. inflating the Eastern 
quota. The presidents of the College Board and ETS are members ex 
officio. 

A commission of *-;crkable size camiot possibly include representa- 
tives of the many groups with a stake in the sprawling area of non- 
traditional study. We have received letters or phoae calls urging the 
appointment of additional members to more adequately represent the 
young, the poor, minorities, women, two-year colleges, secondary 
schools and a variety of professional, accrediting, and te.'^ting interests. 
I expect there are many who have not written or called who have 
concerns equally understandaUw and valid. The Commisi'ion is also 
concerned and plans to establish working relationships with other 
groups and agencies to the extent these are feasible. 

The Chairman, Samuel B. Gould, is chief executive of the operation. 
Chancellor emeritus of SUNY, and is now a director of McKinsey & 
Company. He is spending a substantial fraction of his time on the 
Commission, and has an office in space made available by the College 
Board in New York City. As many of you know, he was chancellor at 
the University of California in Santa Barbara not many years ago. He 
was also for a time head of the £ducrdonal Broadcasting Corporation. 

I am assigned full time as Executive Secretary, as is an Executive 
Assistant, Miss Florence Kiey, and we have secretarial assistance. 



♦Samuel B. Gould, M. Robert Allen, Howard A. Bowen, Mary I. Bunting, 
Henry Chauncey, Arland F. Christ-Janer, Fred C. Cole, Joseph P. Cosand, 
Bertram H. Davis, Walter G. Davis, Frank G. Dickey, W. Todd Fumiss, Richard 
C. Gilman, Cyril O. Houle, Rev. C. Albert Koob, Elizabeth D. Koontz, Charles 
A. LeMaistre, John W. Macy, Jr., Lcland L. Mcdsker, James Parton, James A. 
Perkins, Alice Rivlin, Felix C. Robb, William Schuman, Stephen H. Spurr, 
William W. Tumbull, Clifton R. Wharton, Jr. 
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The Commission will function for two years, with its final recom- 
mendations expected in the fall or winter of 1972-73. It will probably 
hold six meetings, and in addition may sponsor an invitational confer- 
ence, to serve perhaps as a useful sounding board for its initial find- 
ings and recommendations. As n^-Uu^ned above, it has so far met 
once, in Sarasota, on March 8-y, 1971. The -second meeting will take 
place in July, 1971, in the mid-west. The Commission expects to meet 
in various parts of the country, to give expression to its national per- 
spective. 

At its first meeting, the Conunission settled on the following one- 
sentence statement of its charge: 

The Commission will be concerned with increasing access to, 
and recognition of, post-secondary learning by whatever means 
such learning is or could be achieved. 

In organizing itself for its task, the Commission has established six 
subcommittees: 

1) Concepts. One will identify and examine the concepts under- 
lying various facets of non-traditional study. It will seek to clarify 
these in order to make the discussion of issues both within the com- 
mission and in the public at large, more productive. 

One can only guess at this point what the concepts are that this 
subcommittee will focus on. 

A fascinating question despite the many attempts already to answer , 
it is "what is a degree?", or more specifically "what does the American 
baccalaureate represent, in theory and in actual fact?" Where does 
the concept of general education fit in? What is the place of academic 
specialization? Of vocational preparation? Of affective as well as 
cognitive leammg? Of moral and personal growth? 

The Commission at its first meeting talked about the problem of 
placing in proper balance the interests of individuals, institutions, and 
society. This certainly cries for a sound conceptual framework. 

A concept that certainly bears clarification is that of "non-tradi- 
tional study" itself. The Commission is taking an open, flexible posi- 
tion as it moves toward its definition of what is non-traditional. I 
observe it attending to innovative possibilities in the classroom as 
well as outside, on campus as well as off campus, and in processes of 
guidance as well as teaching and certifying. Non-traditional ways of 
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putting together rather traditional forms of study, credit and degree- 
granting seem also to be very much in the ball park. 

Todd Fumiss, of the American Council on EducaUcn, is chairman 
of this subcommittee. 

2) Means. A second subcommittee will focus on the means avail- 
able for teaching and learning at the post-secondary levels, including 
the roles of teacher and student. It will include the residential experi- 
ence, work and community involvements, correspondence, radio, TV, 
cassettes, computers and other technological devices. It will presum* 
ably sort these out, and recommend when and how each can be used 
most effectively. 

James Parton, President of Encyclopaedia Britannica Educational 
Corporation is chairman. 

3) Recognition. Another subcommittee will deal with the recogni- 
tion of post-secondary learning. What are the ways of validating 
learning? What kinds of evidence can and should be used to establish 
that an mdividual has achieved mastery of a subject or skill? What 
are the strengths and limitations of standardized tests? Of faculty 
evaluations? Of biographical and work recprds? How should the bits 
and pieces of evidence be translated into credits? into degrees? What 
are the alternatives to credit bookkeepmg? 

Frank Dickey, Director of the National Commission on Accrediting 
is the chairman. 

Before taking on this new assignment, I was involved for eleven 
years with the Admissions Testing Program of the College Board — 
the Scholastic Aptitude Test and subject-matter Achievement Tests. 
From this experience I bring a sense of confidence in the usefulness 
and values of external examinations, but also questions about their 
applicability on a greatly expanded basis to the post-secondary scene. 

One question is that of the creditability of standardized tests in the 
eyes of college faculty members. For external examinations to bear 
the burden, entirely or partially, of college credits with genuine value, 
their acceptance by large and influential segments of the college teach- 
ing community would seem to be a necessary condition. I doubt if 
this acceptance exists, and I suspect it will not come about easily. 
I sense a considerable lack of communication between those engaged 
in educational measurement and the great majority of college teachers. 
I also sense a ready skepticism or antipathy on the part of many 
college teachers* 
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The resistance I speak of I see as having its sources in emotion as 
well as reason, and in as much misinformation as information. I be- 
lieve, however, that tests for college credit, such as the College Level 
Examinations, may be exposed to a more searching analysis of their 
real nature and relevance than they have so far received, and that 
this may pose difficult but fruitful challenges to testing organizations. 

The problem of coaching or cramming I suspect will come up in a 
variety of new contexts, and will need to be faced. In this connection, 
I think it would be productive to examine the relationships that have 
developed, as tests have evolved, between a set of learning experiences, 
on one hand, and a test devised to compare individuals with respect to 
those experiences. As achievement tests have lost connection with 
delineated, prescribed, published syllabuses, and have sought to com- 
pare fairly individuals whose learning springs from a variety of im- 
plicit "syllabuses," they sometimes adopt the strategy of emphasizing 
whatever is common to all the different syllabuses. The positive 
result Is that the measurement ir reasonably fair, but one negative 
result is that learnings unique to a particular syllabus tend not to be 
measured, and another result is that the test can be coached for on 
the basis of the "least common denominator" elements, which the 
test sets up as a convenient but artificial "syllabus.'' If pressure builds 
to prepare course outlines or reading lists linked with college-level 
tests of this sort, I think it will be difficult to avoid encouraging stu- 
dents to learn what the test happens to test, theieby undermining its 
measurement value, and perverting its educational role. 

Extensive use of college-level tests also may generate new, and I 
believe in this case, beneficial pressures to employ free response as well 
as multiple-choice questions, test results that are more informative than 
standaid scores or percentiles tend to be, and tests that match up 
more sensitively with what those who take them happen to know best. 

4) Access. A fourth subcommittee is concerned with the access 
of individuals to opportunities for study and its recognition. What 
populations are in greatest need of new opportunities? How can in- 
dividuals best be informed about and guided to the opportunities that 
do exist? 

The chairman is Leiand Medsker, Director of the Center for Re- 
search and Development in Higher Education, Berkeley. 

5) Models. A fifth subcommittee is concerned with analyzing and 
making recommendations in regard to models of non-traditional study, 
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particularly external degree models. If one classifies so-called external 
degree arrangements according to who provides the instruction and 
how? who examines the student and how? and who grants the degree? 
with what authority? one quickly generates well over a hundred theo- 
retical variations on the external degree theme. One can find actual 
cases of these models. The crucial question is which model fits best 
the circumstances of a particular student population, particular insti- 
tution, and a particular state. 

Professor Cyril O. Houle, University of Chicago, is the chairman. 

6) Finances. A final subcommittee will be studying thv*5 financial 
implications of all these models and approaches. This suba>mmittee 
will perhaps be particularly useful in comparing the costs of innovative 
approaches with those of more traditional ways of accomplishing the 
same ends. It will give attention to the cost implications for institu- 
tions, for individuals, and for society. 

Howard Bowen, President, Qaremont University Center, is the 
chairman. 

To each of these subcommittees there have been assigned staff 
members of the College Board and ETS. The subcommittees, working 
with these staff members, are now preparing initial papers, with back- 
ground and preliminary recommendations. These will constitute the 
agenda for the July meeting of the Commission, to be held at the 
University of Michigan. 

Fruits. What will be the fruits? The young plant looks healthy and 
vigorous. Its roots are sound and deep. The soil which sunounds it 
feels good. This is the early growing season, however. We will only 
know later on for sure what the Commission has to contribute. 

Some points on which the Commission agreed at its first meeting 
give clues as to the probable nature of the Commission's outcomes: 

The Commission will seek ways of maximizing resources both 
within and without the present system of higher education. 

It will give attention to the matters of standards — what they should 
be, and how they should be applied. 

It will work cooperatively with other interested groups and agencies. 

Above all, it will be action-oriented. 

The Commission takes seriously this action-orientation. It will try 
to see to it that the steps it decides should be taken are taken. There 
will undoubtedly be publications, monographs perhaps, and some 
type of final report, but these are not likely to be the only products. 
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Who needs the Commission? Assuming with hope that the Com- 
mission will facilitate the emergence of an array of possibilities for 
post-secondary learning and credit that offers in its numbers and 
diversity realistic and worthwhile opportunities for all who seek them, 
I think it is clear that those persons for whom such opportunities are 
now lacking, and those educators who are struggling to provide such 
opportunities, will gain from the Commission's work. The assumption | 
is bold, however, and the hope must be joined with imagination, % 
courage, and determined effort by all involved in the Commission's | 
work. I 

I 
i 
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The decade of accountability, at least as I perceive it, started on 
April 11, 1965, when the Congress of the United States passed the 
Elementary and Secondary Education Act, Public Law 89-10. Its 
major thrust was Title I, known colloquially and commonly as com- 
pensatory education or, as we call it, comp ed. Comp ed was the 
nation's recognition that if every child is to receive an equal educa- 
tional opportunity to succeed to his potential, the schools must give 
special attention to the effect that poverty has on the child's learning 
progress. 

Since the enactment of the Elementary and Secondary Education 
Act, which without doubt is the largest single thrust in education in 
the history of this country, billions of dollars have been spent and 
millions of children have been served in one way or another. In recent 
years the critics of this program have been numerous and many times 
the criticisms were justified. However, I do not propose to act as an 
apologist for the success of the program, because I do feel it is success- 
ful, but to trace what I feel lias been a significant impact on the total 
education process in this country. 

The precipitant of this impact was provided under the federal law 
itself, and I think that this is very significant. The U. S. Congress, 
presumably acting for the country as a whole, said this, and I quote: 

Each local educational agency shall, at least annually, provide 
an evaluation of the effectiveness of its program under Title I 
of the Act, in meeting the special education needs of the edu- 
cationally deprived children, including appropriate objective 
measurements of educational achievement. The measurement of 
educational achievement under such a program shall include the 
measuring or estimating of educational deprivation of these 
children who will participate in the program, and the comparing, 
at least annually, of the educational achievement of participating 
children with some objective standard or norm. 

That statement is clear. 
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The Congress also said, "The type of measurement used by a local 
educational a^ncy shall give particular regard to the requirement 
that the State educational agency report to the Commissioner on the 
effectiveness of the programs in that state and proving the educational 
achievement of educationally deprived children/' 

For five years California's Division of Compensatory Education has 
been directly involved in fulfilling that requirement of the law; that is, 
assisting in the construction each year of an annual state evaluation 
report based on the evaluation reports of approximately 800 school 
districts which operate programs that are financed through ESEA, 
Title 1. The strategies used by the districts in fulfilling their obligations, 
particularly in the earlier years — and here I am talking about 1965, 
1966 and 1967 — were indeed wondrous to behold. When we pub- 
lished our first reporting forms, our concept was really quite simple; 
at least it was simple to me. '^Where was the student when he started 
his instructional program? Where was he when he finished his instruc- 
tional program? And what was the increment of gain which, by infer- 
ence, could be attributed to that program?" 

Most districts took quite literally the section of the public law cited 
earlier which we sent to them. They entered into the first phasu of 
evaluating these programs. This phase I shall call the "early quest for 
objectivity." 

While almost all districts have as part of their general educational 
program a process which is euphemistically entitled ''District Testing 
Program," a systematic application of the principles of a sound testing 
program seem to be lost in many of them. Most district testing pro- 
grams have been narrow in scope and spotty in their application. To 
most, objectivity and the reference to some type of normative group 
immediately meant standardized tests, which in a real sense it does. 
Even though many common individual achievement tests existed which 
could be used readily, the search was on for something different, 
because the programs had to be different. 

"Something different" to the districts meant searching out various 
kinds of tests of the most exotic variety, many of which, I think, never 
saw the light of print in Euros* and some of which were completely 
uninterpretable. Indeed, during the first and second years of Title I, 



♦ Buros, Oscar K., Editor, Mental Measurements Yearbook, Highland Park, 
N. J.: Gryphon Press, 1965. 
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we counted 119 different instruments that were used in an attempt to 
assess district programs. 

With almost religious fervor districts set out to construct their own 
instruments^ — ^their own rating scales — ^to assess the nonacademic 
components. As the state educational agency we fostered this quest 
for objectivity to a large extent. In perhaps overinterpreting the legis- 
lative intent of PL 89-10, we asked districts to submit to us samples 
of their questionnaires, opinionnaires, rating scales, or other objective 
assessment devices which they had developed. 

We had to discontinue this practice for a very simple reason. We did 
not have enough space in the house. Some of these devices were rare 
and wondrous, and some of this wondrousness came from the zealous 
nature of some of our staff. 

One example was the reading program. Now, most people know 
how to evaluate a reading program, how to give a reading test, and 
when to initiate a reading testing program; but there are several differ- 
ent ways of teaching reading and there are many different compo- 
nents. In the earlier years, the districts had all this money, and with 
it they entered stages of wildness which took various forms. 

When designing these programs, districts go through predictable 
phases. The first phase is: "How much money do we get, and how 
soon do we get it?" The second phase is: "Now that we have the 
money, what are we going to do with it?" And the third phase, usually 
occurring about 10 months after the start of the program, is: "Oh, 
my gosh, if we have to evaluate, what can we do?" 

I mentioned before the magnitude of the Title I program — billions 
of dollars and millions of children — and the impact on the school dis- 
tricts. But let's put these funds in perspective for a moment. The very 
first assignment I had was to approve the application from the Los 
Angeles Unified School District. And that was the first time I had 
ever signed a piece of paper authorizing the allocation of $19,700,000. 
Los Angles Unified is the largest school district in California so I 
feel I can pick on it. Last year it received approximately $23 million 
in categorical aid funds under Htle I. That is a lot of money. But in 
perspective, it is less than 2 percent of its total operating budget of 
more than $750 million. 

Last year the state of California received approximately $97 million 
for basic Title I programs. This is far less than one-third of 1 percent 
of the total school budget of the state in California. Therefore, we are 
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talking about , sums of money which are abnost unimaginable; yet, 
when we compare those funds to the total education budget, kinder- 
garten through grade twelve in California, the Title I funds seem 
almost puny. The magnitude of Title I funds therefore shrinks and, 
correspondingly, one would expect the impact of such a program to 
be less than it actually is. 

Title I is a highly visible program. And very early in the history of 
the program legislators at the state and national levels again asked a 
very simple question: "Now that you have received your $100 mil- 
lion for the state of California, what gains have these students made, 
or what changes have you seen in the educational program?" Even 
before the ink was dry on the Tide I bill, the lawmakers demanded 
to know the impact of he program, not only nationally and at the 
state level, but in mary, many instances at the local level. 

And this led to the second phase of evaluating Title I programs 
which I shall call "the quest for significance." Again, aided and abetted 
by a zealous staff at the state level, districts — even though th^ had 
not answered fully the basic question of how far do the children go — 
were now intent on proving the statistical significance of their findings. 
So we got bundles, truckloads of [sic] tests, analyses ''of variance, 
analyses of covariance, and other fancy statistical techniques. 

It has been proven, I am sure at least several hundred times, that 
the incremental gain of the post test over the pretest for several hun- 
dred students is at or beyond the one percent level of confidence. These 
gains were duly noted and reported. I knew these statistics had little 
meaning and the districts, I hope, knew they had little meaning. When 
you test 500 children, just by living, they are going to have a change 
in their test scores. 

I guess our communications were poor. The legislative bodies asked 
again, "Tell us, simply — ^you know, don't give us all this garbage — 
how much gain did you make?'' There is only one statistic that the 
lawmaking bodies understand, and that is the ubiquitous grade equiva- 
* lent. Then they asked the question for the third time — it was in a legis- 
lative hearing that I first heard the phrase that I have come to abhor: 
"How much bang did you get for the buck?" We are indebted to Mr. 
McNamara for that phrase. Most everybody in the whole world asso- 
ciates the F-111 with Mr. McNamara, but I associate "bang for the 
buck" with him. 

To answer these insistent and, frankly, very logical inquiries, we 
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entered into our next phase of Title I evaluation. This is the ''phase of 
accountability," the shibboleth which you are condemned to hear for 
the next decade at least. 

Now, the accountability concept did sot emerge whole bom; it 
went through evolutionary phases. Those evolutionary phases could 
be characterized by a series of questions: How much does it cost to 
increase the child's reading achievement by one year? What program 
components are most effective in promoting achievement in the basic 
skills? These are followed by the amalgam: Which components are 
cost effective? And this is how we arrive at the "bang for the buck" 
component. 

I am sure I am taking too parochial an attitude when I attribute 
the current concept of accountability to categorical aid programs. You 
will note that these are questions the lay community has been asking 
for a long time, questions never satisfactorily answered by the edu- 
cational community. Just look at the failures in the bond issues. Now, 
I am sure that a very high proportion of these failures can be due to 
the tax base, but I think above and beyond this it is a failure of com- 
munication between the educational community and its constituents. 

Five years ago we instituted for a fair proportion of the school 
population a systematic design for the collection of achievement data. 
As districts became more sophisticated with their techniques of data 
collection and analysis, not only were the innovative curricular aspects 
generalized to the total school program — and we have good documen- 
tation of this now — districts also found that there were many general 
applications of the evaluative techniques which we had coerced them 
into. 

In the past, data from regular school testing programs, now con- 
sidered necessary for curricular modification, were generally not 
gathered in a systematic fashion, or the educational community did 
not communicate results of the program to their constituents. Reports 
were given to satisfy the boards of trustees that each district as a 
whole was average. Everybody is happy when things are average. 
The boards of trustees were told that their districts were average, 
while in fact 20 to 40 percent of their schools were substandard. But 
this 20 to 40 percent were buried, you see, becaus*; they were all 
aggregated together and the golden mean emerged. The low-achieving 
children in these schools had a dubious status — they were poor or 
black or brown — and thus everyone expected them to be low achievers* 

45 



Mandated Evaluation 



We rejected this construct, and we substituted our own hypothesis: 
that students of whatever ethnic origin have a distribution of ability 
which is normally distributed and whose mean probably is not differ- 
ent, given adequate educational and environmental opportunities, than 
that of the population as a whole. 

We have a substantial body of evidence, based primarily on the 
early years of education, that disadvantaged children can, in fact, 
achieve at "grade level'* on the average and that their achievement 
scores are symetrically distributed. The critics of compensatory edu- 
cation are many and say that the schools have not provided the im- 
petus nor shown the gains nor lived up to the original expectations 
of the Elementary and Secondary Education Act. I reject this general- 
ization. 

Our advocates also are many, and the evidence is, at least from 
my rather provincial viewpoint of California, ratlier preponderant, 
that, given the kind of program which we have outlined in compensa- 
tory education, we can and have moved groups of students to the 
point where, as one example, the level of achievement of an entire 
graduating class in a Title T target school was actually superior to the 
average achievement scores of students in the district as a whole. 

Programs that are effective were discovered through the use of 
systematic evaluation. We have attempted to generalize these programs 
to other school populations. Sometimes we were successful and some- 
times we were not, because obviously one cannot transfer the enthusi- 
asm of the teacher and the motivation of an administrator or support 
of a parent group from one school or district to another school oi 
district. 

The age of accountability is, in the most general sense, I believe, 
responsible for program effectiveness. Mandated evaluation and the 
development of a systems analysis approach to education is part of 
the package of accountability. The development of PPBS is the start 
of a rational accounting system. PPBS refers to Program Planning and 
Budgeting System, which is sometimes called PPBES in which E 
represents evaluation or sometimes called management information 
systems. The system has a variety of acronyms depending on who is 
the sponsoring agent, such as ASBO, the state of Colorado, or the 
state of Oregon, but they all mean about the same thing. And here 
I mean that it is a system of accounting in the broader sense of the 
word. 
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As for the development of behavioral objectives, we have been in- 
quiring about it for several years in Title I, and education is really 
rediscovering the wheel. We tend to do that about every other decade. 
We have to rediscover something that was discovered perhaps 20 
years before. 

I was first exposed to performance objectives or behavioral objec- 
tives, whatever you wish to call them, during my undergraduate work 
in psychology in 1950. 1 feel it is appropriate here to get in my jousts 
at the windmill of criterion-referenced tests, which are a corollary to 
performance objectives. I don't believe so-called criterion referenced 
tests are indeed different Trom norm-referenced tests. Yes, it is easy 
to make a semantic distinction between them, and yet on application 
of these instruments, one finds in the last instance, there is inevitably 
a normative type of interpretation. And whatever the process used is, 
the interpretation is the final message. Be it norm or criterion refer- 
ence, the message comes out that performance is better than or worse 
than, greater than or less than, or different from some other population 
or sample. We find that the northeastern quadrant is better than the 
southeastern quadrant. And all these, I submit, are normative state- 
ments. So I really see a false distinction between these types of tests. 
Both are useful and both deserve a separate label. But I think we 
should be very cautious in our interpretation of them. 

I am sure that accountability concepts would have appeared on the 
educational scene without the impact of categorical aid funds. But 
it is my belief that ESEA, Title I, hastened the arrival of account- 
ability concepts by mandating systematic evaluations, encouraging 
experimentation in curriculum, involving parents and community 
members in decision making based on information derived from pro- 
grams, making an annual educational audit (Title I did it, Lessinger 
named it) and asking districts to set up measurable objectives. 

Several years ago Marvin Alkin, in addressing this conference, 
issued a caveat regarding "behavioral objecti«^es." I have a few caveats 
about evaluation. Mindful of the educational pendulum, I am a little 
afraid of the amplitude of the arc of accountability and all that has 
been detailed. I sense a glut of ill-wijtten objectives, with equally bad 
items purporting to measure the outcomes of these objectives. Good 
objectives can be written, comprehensive and realistic evaluations can 
be made, and effective communications can be established. I have 
seen all of this accomplished. 
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Just over a hundred years ago Georgia was virtually destroyed by the 
devastations of the Civil War. However, Georgia managed to rebuild, 
like the phoenix who rises again in youthful freshness after being 
consumed in fire. 

As did the phoenix of classical antiquity, Georgia returned to its 
nest to be consumed by the flames of the dual school system. From 
the pyre of the disestablished dual educational system a young phoenix 
is being born and the opportunity for educational reform has emerged 
mih youthful freshness and vigor. 

I am here to tell you about a low profile operation in the Georgia 
Department of Education— the Division of Planning, Research and 
Evaluation — and one of its projects that is capitalizing on the vigor 
and freshness of this new education opportunity. This project is the 
Georgia Assessment Project — GAP. 

GAP is a project designed to measure the quality of education on a 
statewide basis. Results mU be used for: 

(1) measuring the impact of educational programs, services and 
resources on children and youth 

(2) determining the relationship between costs and educational 
benefits 

(3) identifying areas of critical educational need, and 

(4) developing long-range educational planning. 

GAP is just a part, albeit a very important part, of the Division's 
over-all efforts. The flow chart (Figure 1) broadly outlines the Georgia 
Department of Education's plan to deliver the state-level resources for 
elementary and secondary education based on identified student needs. 
I will be talking about the process we went through in goal-setting as 
reflected in Panels 1 through 5, and the preparations now underway 
for measuring pupils* progress toward the goals as reflected in Panels 
6 through 8. 

The Georgia Assessment Project is based on a series of assump- 
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tions. I would like to share with you some of these assumptions and 
the rationale for each. 

Our society is increasingly characterized by change. Many changes 
will have important implications for us and will necessitate changes in 
education. GAP recognizes that there is an urgent need to anticipate 
the changes that are likely to occur in society during the next 15 
years, to understand their implications for education and to plan the 
necessary adjustments to meet emerging needs. Some of the prospec- 
tive changes may be beneficial to society; others may be harmful. 
Man, to some extent, can control the nature and direction of change. 
With increased knowledge and understanding he should be in a better 
position to plan and prepare for those changes that are beneficial and 
to avoid those that could be disadvantageous or even disastrous. We 
also are preparing students to live in the society of the future. If we 
know what the society might be like and what it ought to like, our 
education programs can provide a positive force in seeking and effect- 
ing meaningful and desirable social, political and economic change. 

Goal Directed ^ 

The education effort can be evaluated maturely and defensibly only 
in light of the ends it seeks to secure. Goal statements abound in edu-- 
cation. From the Committee of Ten in 1894, through the Seven Car- 
dinal Principles of 1918, to the statements of the Education Policies 
Commission in recent times, educators and others have formulated 
goals. Such statements are usually good; they are fundamentally in 
harmony — indeed, often quite similar; and yet they do not seem to 
satisfy because they are so broad, so general. 

Fundamental to GAP, then, is the thesis that an initial step in pro- 
viding a measurement of education is to set goals for education that 
identify the qualities and characteristics which citizens should possess 
if they are to live self-actualizing lives iu the future. 

State Board to Set Goals ^ 

The State Board of Education in Georgia is vested with vast power, 
authority and responsibility with respect to the education enterprise. 
More than sixty-five percent of all funds for public elementary and 
secondary education are managed by the State Board of Education. 
The Board is placed in a position where every citizen in the State 
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can observe its actions. Because of this elevated position, it is able to 
view the State and its citizenry, from the largest city to the smallest 
hamlet and it is therefore concerned with the quality of education 
available and the educational accomplishment of every man and 
woman, every boy and girl. Furthermore, it is composed of lay citizens 
from all geographic areas of the State. The very existence of the Board 
reflects the time-worn principle of public education that, after all is 
said and done, the schools still belong to the people. 

GAP recognizes that because of the Board's vast authority and 
management responsibility, statewide perspective, geographic repre- 
sentation and lay composition, it is appropriate and, yes, even man- 
da\ory not only for the Board to generate its own goals for education, 
but also to provide for the statewide measurement of education pro- 
grams which seek to implement and achieve these goals. 

Education and the Larger Society 

A central concept in American public education is that it derives its 
purpose, form and content from the particular social environment in 
which it exists. Education is a product of the culture of which it is a 
part. Consideration must be given to such matters as the worth and 
dignity accorded the individual, religious ideals, the sources of political 
power, the class structure, the nature and operation of the economy 
and the thought patterns of the age, because all are woven at any given 
time and place into the purpose and form of the education enterprise. 

GAP, recognizing that education is only a part of the larger society 
and that the schools belong to the people, concluded, that in order 
to provide assistance to policymakers in setting goals for education it 
was appropriate to turn to outstanding citizens who might be in a 
better position to view society and culture in its entirety. 

Focus on the Products of the Education Process 

Past and present accountability and assessment efforts far too often 
focus on the inputs or processes of the education endeavor rather than 
the product of education — the child, youth or adult. If learning is 
defined as change in behavior and if we are to make an adequate 
assessment of the education enterprise, we must then look at the 
child. Processes and inputs become important only when related to 
behavioral changes in students. 
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GAP is committed to assessing die progress of education by looking 
at the productivity of the schools. 

Expansion of Goals into Performance Objectives 

Few well-informed educators have not heard of behavioral objectives. 
The force behind this resurrected term seems to be the recognized 
need to express desired learning outcomes as specifically as possible 
so that progress toward achieving these outcomes may be easily 
measured. 

GAP recognizes that the goals for education should be stated in 
such specific terms and in such concrete language that the goals the 
students achieve are readily apparent to anyone observing students' 
behavior. Therefore, each goal will be expanded into performance 
objectives expressed in measurable terms. 

Construction of Criterion-Referenced 
IMeasurement Exercises 

In the past, when student outcomes have been measured, we have 
relied almost exclusively on standardized tests. Standardized tests 
have been used primarily because they are cheap, quick and readily 
available. However, standardized tests are normative in nature, de- 
signed to ascertain an individual's performance in relation to the per- 
formance of other individuals measured by the same device. They do 
not provide an adequate measurement of student performance in 
terms of the extent to which he has mastered a given set of objectives 
without reference to the performance of any other student.^ Criterion- 
referenced measures, on the other hand, are used to ascertain an 
individual's status with respect to some criterion. They make no 
comparisons among individuals. 

GAP will focus on the preparation of criterion-referenced measure- 
ments so that the performance of students may be related to behavioral 
objectives, which in turn are a more specific expression of student 
performance toward acquiring those qualiiies and characteristics 
sought in the goals of the State Board of Education. 

Sampling Procedures 

Perhaps the most useful and versatile tool available to researchers is 
sampling. Modem sampling techniques using only five percent of the 
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study population make possible many studies such as GAP that could 
not be done on a total universe basis. Moreover, sampling often makes 
data available more quickly, more economically and sometimes more 
accurately than do miiverse studies. 

Tentatively, GAP will focus initially on three age levels: 9, 13 and 
17. These ages seem to be the most plausible, for at 9 children 
have been exposed to the basic programs of primary education; at 13 
most have concluded elementary school education; and at 17 youth 
are close to completing their secondary school education. Only a small 
percentage of children at each age level will need to be involved ia 
GAP. 

In addition, to determine the educational progress of children at 
various age levels, GAP thought it important for education decision 
makers to have information about certain student sub-populations. 
The tentative dimensions for sub-populatioa stratification are sex, 
race, regions of the State, type of conmoiunity the child lives in and 
socioeconomic background. 

It appears that the number of measurement exercises for each goal 
may be extensive. Therefore, exercises as well as student samples 
are to be selected concurrently, using a multi-matrix sampling pro- 
cedure. The age levels and some of the sub-population strata, as you 
may recognize, are not too dissimilar to those of the National Assess- 
ment of Educational Progress. We hope to use some of the NAEP 
exer*. es in GAP. This will give us regional as well as national com- 
parability of student performance. 

Reporting Results 

Since the lay public and its representatives, the members of the Legis- 
lature, are demanding to know more about educational progress, 
these arc the groups to which the results must be communicated in 
as easily an undeistandable manner as possible. In addition, more 
detailed reports will be prepared for distribution to professional edu- 
cators, so that these decision nakers too may have better information 
to guide decisions. 

The important point is that everything with respect to student per- 
formance will be above board. Information about shortcomings as 
well as accomplishments will be made available to all. In this way we 
hope to avoid any criticism of data withholding or manipulation of 
results. 
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Identification of Critical Educational Needs 

The results of GAP, as they become available, should provide a 
quantitative assessment of student performance as related to the Goals 
for Education in a wide range of areas where the State education enter- 
prise has done weli and v/here it has not done well. The delivery of 
State funds and State Department of Education services can then be 
directed to meet the idenUiied areas of critical educational need to 
v/heve the discrepancy between actual performance and desired per- 
formance has been ideatiiied. 

Use in Local Sciioot Systems 

It is anticipated that, a^ the GAP model is completed and tested and 
the initial assessment is made, the expertise gained by the GAP staff 
on a statewide S^/^^; m^y be used to provide technical assistance and 
services to local school sy:Uems so they too may conduct their own 
assessment. ScvCial school systems already have used either a mod- 
ification of the GAP process for goal setting or the GAP goals them- 
selves for their own planning, assessment and accountability efforts. 

instruments Used for Other Purposes 

The instruments developed in GAP will have other applications. One 
of these will be to conduct studies to help determine what effect inputs 
into the education enterprise have on student performance. As greater 
sophistication in simulation techniques is gained, cost-effective analy- 
ses can be used to simulate education decisions so that we need not 
wait for years and years to find out what results might be expected 
vAien certain policies are adopted. 

The PInases of GAP 

I would like to share with you briefly each phase and major activity 
of GAP. 

Pliase I — Setting Goals 

In 1969 the Stat; Board created an Advisory Commission on Edu- 
cation Goals made up of eleven distingiiished citizens of the State. 
The tasks of the Commission were to: 

1. Examine the social, economic and political life of Georgia 
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2. Project the probable social, political and economic conditions 
of the State through 1985 

3. Identify as **goals for education," the knowledge, skills and 
values that will enable the citizen of Georgia to successfully live 
in and shape the future, and 

4. Suggest the nature of the education system necessary to achieve 
the desired goals. 

Recognizing fully that the Commission was not endowed with the 
ability to gaze into a crystal ball, highly qualified specialists prepared 
twenty position papers about Georgia's current status and its probable 
status in 1985 with respect to the social, economic, technological, 
political and cultural environment. Some of the areas examined were 
Economy, Manpower and Employment, Social Disorganization, Re- 
ligion, the Structure of Government, Ecology and the Arts. 

In addition, 27 critiques were prepared by other specialists to pro- 
vide additional analyses, corrections and amplifications. The papers 
and their critiques have been published in a book. Focus on the Future 
of Georgia, 1970-1985, Edited by William H. Schabacker, Russell 
S. Clark, and Homer C. Cooper.^ In addition to the book, a film, 
"The State of the Future," has also been prepared. It has been shown 
on the Georgia Education Television Network and has been used by 
numerous non-education and education audiences. 

Conditions and consequences of the emerging society of Geor^a 
were phrased from the papers and critiques in the form of propositions 
by the GAP staff to help the Goals Commission better understand 
what the future might be like. 

The Commission was asked six questions about each proposition: 

1. What is good about this condition and what is bad? 

2. What can be done to perpetuate the good? 

3. What can be done to rectify the bad? 

4. Is it desirable and necessary for the education enterprise under 
direction of the State Board of Education, to seek and provide 
educational experiences that would develop these qualities in 
each individual? 

5. What qualities should the citizen possess that would promote 
the well-being of everyone and assure individual worth and 
dignity? 

6. Fmally, what qualities should the citizen possess that would 
enable him not only to live successfully in the environment of 

55 



Georgia Assessment Project 

1985 and beyond but also to shape that society? 
As answers to these questions were formed, the Goals Commission 
prepared the recommended ''Product Goals" that described the quali- 
ties and characteristics each individual should possess that, upon 
leaving the secondary school would help him become a person pre- 
pared for many options. The Product Goals are classified under the 
seven headings: 

1. The Individual and Himself 

2. The Individual and Others 

3. The Individual and the Governing Process 

4. The Individual and Social and Economic Institutions 

5. The Individual and His Physical Environment 

6. The Individual at Work 

7. The Individual at Leisure 

Then the Goals Commission suggested the means — the inputs and 
processes — that would help in achieving the Product Goals. The 
means, called ''Enterprise Goals" by the Commission, are classified 
under six headings: People to be Served, The Curriculum, The Staff, 
Organization and Administration, Buildings and Facilities, and Fi- 
nance. Since the GAP effort is concerned only with measuring pupil 
performance with respect to the Product Goals, the Enterprise Goals 
have no immediate value to GAP. However, in the flow chart (Figure 
1) the Activities labeled 10, 11 and 12 relate to the preparation and 
assessment of a long-range program structure and program objectives. 
The project within the Division of Planning, Research and Evaluation 
concerned with these activities has immediate use for the Enterprise 
Goals. 

The goals were then submitted to the State Board. After reviewing 
the report the Board adopted the Commission's recommendations. The 
full text of the report as well as the Product Enterprise Goals have 
been published in a booklet. Goals for Education in Georgia.^ 

Phase II — ^Development of Objectives and Measurement Exercises 

As the first step in Phase II, a group of knowledgeable people fa- 
miliar with various aspects of Georgia life will be identified. Each per- 
son will be asked to take each product goal that might relate to his 
area of expertise and prepare a list of descriptions of observable be- 
havior for a person about to graduate from high school that would 
show his acquisition of the qualities and characteristics called for in 



56 



William Schabacker 



that goal. From these lists the GAP staff will write performance ob- 
jectives expressed in measurable terms for each of the product goals. 

The objectives will then be submitted to panels of lay persons. 
Based on its plausibility as a valid performance expectation related to 
the qualities and characteristics sought in the product goals, the panels 
will be asked to accept, reject or revise each performance objective. 

The approved performance goals will then be submitted to a group 
of teachers and curriculum specialists. These professional educators 
will be asked to determine at which age level stratum of the pupil 
population the students should be able to accomplish each particular 
performance objective. If the expected performance is for the age 17 
or 13 level, these specialists will then be asked to determine the 
behavior expected at the age 9 level that would show progress toward 
accomplishment of the performance objectives appropriate for the 
17-year-old. 

The output of the professional educators will then be submitted to 
other panels of lay persons. For each age level these panels also will be 
asked to accept, reject or revise each performance objective based 
on its plausibility as a valid performance expectation, i.e., whether it 
shows progress or achievement of the qualities and characteristics 
sought in the product goals. 

Criterion-referenced exercises that will provide the method of 
measuring progress toward what is sought in each goal will be con- 
structed by the GAP staff with the advice of measurement specialists. 
Delivery systems for the exercises will then be prepared. At this point 
we arc leaving all options open with respect to delivery systems. Such 
technological resources as the statewide educational television net- 
work, video tape, recorders, films, film strips, tape recorders, as well 
as traditional paper and pencil tests, all hold some promise as poten- 
tial delivery systems for the measurement exercises. The exercises 
and delivery systems will then be field-tested. It is anticipated that 
this phase. Phase 11^ will take two years. 

Phases III and IV 

Phase III will be carrying out the initial assessment using the 
sampling procedures previously described. The exercises will be ad- 
ministered to the sample using the delivery systems developed in 



Phase IV wiU determine the areas of critical educational need. As 
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the measurement results are collected and analyses made, areas of 
critical educational need may then be identified. The needs will be 
defined as the gap between ^^at minimal qualities and characteristics 
the society of Georgia seeks in all students — the product goals for 
education — and how well students perform on measurement exercises 
designed to determine progress toward these goals — the actual per- 
formance level as determined by the assessment. 

In Conclusion 

GAP as a project will be concluded when the reporting phase has 
been completed. If it is determined that assessment should become 
an ongoing program of the State Department of Education, the re- 
cycling process will begin as measurements are taken from time to 
time and pupil progress is reassessed. The GAP-developed instruments 
will be available to conduct cost-effectiveness studies that can relate 
input to results. Resources can then be mobilized to provide whatever 
is necessary to narrow the gap between actual student performance 
and what society seeks from the education enterprise. Other studies 
using the GAP-developed instruments can be used to determine what 
effect life outside the school may be having on student performance. 

It is possible that we will then be one step closer to getting some 
answers to the often asked question, "When you take X dollars and 
muhiply it by X children with this kind of need, what kind of resuh 
can you expect?" Hopefully we can determine what educational pro- 
grams are having a good effect on children, what are having a bad 
effect and what are having no effect at all. 

Using the results of GAP — ^the data on student performance, cri- 
terion-referenced instruments and field-tested delivery systems — the 
Georgia Department of Education expects to be able to capitalize on 
the opportunity for education reform. Using strategies designed to 
assure results we expect to be in a position to mobilize the total re- 
sources of the statewide education effort toward areas of critical edu^ 
cational need. The State Board of Education will then be in a position 
to be fully accountable for its decisions regarding the total pubUc 
elementary and secondary education enterprise in Georgia. 
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